A large number of enterprises today have multiple satellite offices. With increasing globalization, those locations span multiple time zones, range from small offices of less than twenty users, to large facilities of several thousand users. Recently, there has been a move towards centralized data management for such enterprises. Centralizing data resources leads to reduced infrastructure and personnel cost at satellite locations (clients), and resource consolidation at the data center (server) side. Data centers typically employ well trained professionals, have better recovery and disaster management support, and can provide centralized policy enforcement.
Deployment of such data management architecture for remote backups and disaster recovery is common. There is, however, a need for a network file system for interactive read-write data access in the wide area network (WAN) setting. Traditionally, network file systems have been designed for local area networks, where bandwidth is ample and latencies are low. Common networked file systems like NFS and CIFS transfer large amounts of data frequently. All writes are transmitted to the server and require synchronous updates to the files there. Apart from wasting bandwidth, typical networked file systems require multiple round trips to complete a single file operation. The metadata requests are synchronous and the client cannot proceed without receiving a server response. The high latency of the round-trips over the WAN and the “chatty” nature of the protocols make file access slow and unreliable. Finally, relying on a central server over the wide area network makes the file system susceptible to significant slowdowns due to unpredictable network delays and outages.
Networked file systems have been an active area of research. In an early system, clients transmitted all system calls, e.g. open, close, read, write, etc., to the server. Network file systems have been studied in the local area with stateless and stateful servers. Recently, there has been significant research activity in providing data access (object or file system based) over the WAN. Multiple peer-to-peer architectures for decentralized data management have been proposed. The goal of such systems, however, is to store large quantities of data, dispersed and replicated across multiple clients to improve fault resilience and reduce management overheads. In contrast, the systems and methods of the present invention improve performance of existing network file systems for interactive workloads. While a system of the present invention is capable of storing large data, replication, and disconnected operation, such characteristics are not the primary concern.
Independently, improving the performance of large file downloads in overlay networks has also been studied. Those systems target client downloads of whole data objects like movies, software distributions, etc., from one or more publishers. They do not maintain object hierarchies like directories, and do not consider modifications to objects. An entirely different workload is targeted by the system of the present invention. For example, the table of FIG. 12 shows the distribution of the different NFS RPC calls in a trace collected in an existing system. From the distribution of the RPC calls, it is clear that a significant portion of the network communication is due to the lookups and other metadata traffic. In a WAN environment, such communication imposes a significant overhead on the performance of the file system. Previous efforts to provide wide area file system access optimize mainly for the bandwidth. Reducing the latency of those metadata transfers is a primary design goal of the present invention in addition to providing high-bandwidth parallel downloads.
Using interposing proxies along with caching and parallel downloads, the systems described by A. Muthitacharoen, B. Chen, and D. Mazieres, A Low-Bandwidth Network File System, in SOSP '01: Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, pages 174-187, 2001, and by S. Annapureddy, M. J. Freedman, and D. Mazieres, Shark: Scaling File Servers via Cooperative Caching, in Proc. of 2nd Usenix Symposium on Network Systems Design and Implementation NSDI '05, Boston, Mass., May 2005, have been shown to reduce the bandwidth utilization of file system transfers. However, the techniques described target only data transfers, and are either specific to a single client-server pair or do not directly enable sharing across multiple client sites. The former system is primarily designed for a single client-server pair. The size of the content cache and the eviction policy do not significantly hinder performance. The latter uses geographically distributed cooperative caching proxies that enable fast parallel downloads in addition to difference elimination and content caching. That system is designed for environments where multiple clients are interested in the same large file concurrently; e.g., for an experiment. Therefore, no attempt to improve the metadata access performance as well as inter-site read-write sharing is explored.
There is therefore presently a need to provide a high-speed networking solution for WAN-based file system architecture. To the inventors' knowledge, no such solutions are currently available.